# Low-Latency Inference

NeuroBERT-Mini is a lightweight natural language processing model derived from google/bert-base-uncased, optimized for real-time inference on edge and IoT devices.

A multilingual audio classification model based on speechbrain/lang-id-commonlanguage_ecapa, supporting identification of 5 Indian languages

This is a very compact and fast Japanese reranking model, suitable for improving the accuracy of RAG systems and can run efficiently on CPUs or edge devices.

This is a very compact and fast Japanese reranking model, suitable for improving the accuracy of RAG systems.

FP8 quantized version of Qwen2.5-VL-72B-Instruct, supporting vision-text input and text output, optimized and released by Neural Magic.

Gemma 3 4B parameter model optimized with OpenVINO, supporting text-to-text and visual-text inference

Distil-Whisper is a distilled version of the Whisper model, optimized for Automatic Speech Recognition (ASR) tasks, offering faster inference speeds.

A CTranslate2 format model converted from Distil-Whisper large-v3.5 for efficient speech recognition

The RWKV-7 model adopts the flash linear attention format, supports multilingual text generation tasks, and has a parameter count of 2.9 billion.

Phi-4-multimodal-instruct is a lightweight open-source multimodal foundation model that supports text, image, and audio inputs to generate text outputs, with a context length of 128K tokens.

INT8 quantized version based on mgoin/pixtral-12b, supports vision-text multimodal tasks with optimized inference efficiency

Quantized version of Qwen2.5-VL-7B-Instruct, supporting vision-text input and text output, optimized for inference efficiency through INT8 weight quantization

The LB Reranker is a model for determining the relevance between queries and text snippets, supporting 95+ languages, suitable for ranking and reranking in retrieval tasks.

Kotoba-Whisper-Bilingual is a distilled model collection trained from the Whisper model, specifically designed for Japanese and English speech recognition and speech-to-text translation tasks.

An improved model based on Llama2 7B, supporting hierarchical skip and self-speculative decoding to enhance inference efficiency

FireFunction V2 is a state-of-the-art function calling model with a commercially viable license, trained on Llama 3, supporting parallel function calls and strong instruction following.

LLM Compiler is an advanced LLM based on Code Llama, specifically designed for code optimization and compiler reasoning tasks

TinyAgent is a small language model (SLM) designed for edge devices, focusing on function calling and complex reasoning capabilities, providing privacy protection and low-latency services.

Hiera is a hierarchical vision Transformer model that is fast, powerful, and concise. It surpasses state-of-the-art performance in a wide range of image and video tasks while significantly improving runtime speed.

CodeGemma is a lightweight open-source code model series based on Gemma, specializing in code generation and dialogue tasks.

A German speech recognition model based on distil-whisper technology, with 756 million parameters, achieving faster inference speeds while maintaining high quality.

The Ragas Evaluation Model is part of the Ragas synthetic test data generation pipeline, serving as an alternative to GPT-4 for evaluation tasks.

Distil-Whisper is a knowledge-distilled version of Whisper large-v3, focusing on English automatic speech recognition, offering faster inference speeds while maintaining accuracy close to the original model.

This is a version of the distil-whisper/distil-medium.en model converted to CTranslate2 format for efficient speech recognition tasks.

This is a distilled version of the automatic speech recognition (ASR) model based on the Whisper architecture, designed for efficient inference and suitable for English speech-to-text tasks.

This is the quantized version of multilingual-e5-small, optimized for inference performance through layer-wise quantization while retaining most of the original model's quality.

This is the ONNX format conversion of the papluca/xlm-roberta-base-language-detection model, designed for multilingual text classification tasks, supporting detection in 20 languages.

A 3.3B-parameter causal language model specialized in code completion tasks, supporting 30 programming languages

Quantized (INT8) ONNX variant of BGE-large-en-v1.5 with inference acceleration via DeepSparse

This is an OpenVINO-optimized version converted from MIT/ast-finetuned-speech-commands-v2, designed to accelerate inference operations for voice command recognition tasks.

EfficientFormer-L3 is a lightweight vision Transformer model developed by Snap Research, optimized for mobile devices to achieve low latency while maintaining high performance.

MobileNet V1 is a lightweight convolutional neural network designed for mobile devices, balancing latency, model size, and accuracy in image classification tasks.

MobileNet V1 is a lightweight convolutional neural network designed for mobile and embedded vision applications, pre-trained on the ImageNet-1k dataset.

MobileNet V2 is a lightweight vision model optimized for mobile devices, excelling in image classification tasks.

A lightweight image classification model pre-trained on the ImageNet-1k dataset, specifically optimized for mobile devices

OpenVINO IR format version of the T5-small model, supporting text generation, translation, and other tasks

MobileNet V2 is a lightweight convolutional neural network designed for mobile devices, excelling in image classification tasks.

MobileNet V2 is a lightweight convolutional neural network designed for mobile devices, excelling in image classification tasks.

MobileNet V1 is a lightweight convolutional neural network designed for mobile and embedded vision applications, pretrained on the ImageNet-1k dataset.

A cross-encoder model trained on the MS Marco passage ranking task for query-passage relevance scoring in information retrieval.